NPS-LM-04-025 


EXCERPT  FROM  THE 

PROCEEDINGS 

OF  THE 

FIRST  ANNUAL  ACQUISITION 
RESEARCH  SYMPOSIUM 


CHARACTERISTICS  OF  GOOD  METRICS  FOR  PERFORMANCE 

BASED  LOGISTICS 

Published:  30  September  2004 
by 

Kenneth  Doerr,  Donald  R.  Eaton,  RADM,  USN  (ret.),  and  Ira  A.  Lewis 

2nd  Annual  Acquisition  Research  Symposium 
of  the  Naval  Postgraduate  School: 

Charting  a  Course  for  Change: 

Acquisition  Theory  and  Practice  for  a  Transforming  Defense 

May  13,  2004 


Approved  for  public  release,  distribution  unlimited. 

Prepared  for:  Naval  Postgraduate  School,  Monterey,  California  93943 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Form  Approved 
OMB  No.  0704-0188 


Report  Documentation  Page 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 


1.  REPORT  DATE 

13  MAY  2004 

2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2004  to  00-00-2004 

4.  TITLE  AND  SUBTITLE 

Characteristics  of  Good  Metrics  for  Performance  Based  Logistics 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Postgraduate  School, Graduate  School  of  Business  and  Public 

Policy ,555  Dyer  Road,  Room  332, Monterey, CA, 93943 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

1st  Annual  Acquisition  Research  Symposium:  Charting  a  Course  for  Change:  Acquisition  Theory  and 
Practice  for  a  Transforming  Defense,  13  May  2004  in  Monterey,  CA 

14.  ABSTRACT 

see  report 


15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

18.  NUMBER 
OF  PAGES 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

24 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  research  presented  at  the  symposium  was  supported  by  the  Acquisition  Chair  of 
the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval  Postgraduate  School. 


To  request  Defense  Acquisition  Research  or  to  become  a  research  sponsor, 
please  contact: 

NPS  Acquisition  Research  Program 
Attn:  James  B.  Greene,  RADM,  USN,  (Ret) 

Acquisition  Chair 

Graduate  School  of  Business  and  Public  Policy 

Naval  Postgraduate  School 

555  Dyer  Road,  Room  332 

Monterey,  CA  93943-5103 

Tel:  (831)656-2092 

Fax:  (831)656-2253 

E-mail:  ibqreene@nps.edu 

Copies  of  the  Acquisition  Sponsored  Research  Reports  may  be  printed  from  our 
website  www.acquisitionresearch.org 

Conference  Website: 
www.researchsvmposium.org 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Proceedings  of  the  Annual  Acquisition  Research  Program 

The  following  article  is  taken  as  an  excerpt  from  the  proceedings  of  the  annual 
Acquisition  Research  Program.  This  annual  event  showcases  the  research  projects 
funded  through  the  Acquisition  Research  Program  at  the  Graduate  School  of  Business 
and  Public  Policy  at  the  Naval  Postgraduate  School.  Featuring  keynote  speakers, 
plenary  panels,  multiple  panel  sessions,  a  student  research  poster  show  and  social 
events,  the  Annual  Acquisition  Research  Symposium  offers  a  candid  environment 
where  high-ranking  Department  of  Defense  (DoD)  officials,  industry  officials, 
accomplished  faculty  and  military  students  are  encouraged  to  collaborate  on  finding 
applicable  solutions  to  the  challenges  facing  acquisition  policies  and  processes  within 
the  DoD  today.  By  jointly  and  publicly  questioning  the  norms  of  industry  and  academia, 
the  resulting  research  benefits  from  myriad  perspectives  and  collaborations  which  can 
identify  better  solutions  and  practices  in  acquisition,  contract,  financial,  logistics  and 
program  management. 

For  further  information  regarding  the  Acquisition  Research  Program,  electronic 
copies  of  additional  research,  or  to  learn  more  about  becoming  a  sponsor,  please  visit 
our  program  website  at: 

www.acquistionresearch.org 

For  further  information  on  or  to  register  for  the  next  Acquisition  Research 
Symposium  during  the  third  week  of  May,  please  visit  our  conference  website  at: 

www.researchsvmposium.org 


CHARTING  A  COURSE  FOR  CHANGE: 

ACQUISITION  THEORY  AND  PRACTICE  FORA  TRANSFORMING  DEFENSE 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


CHARTING  A  COURSE  FOR  CHANGE: 

ACQUISITION  THEORY  AND  PRACTICE  FORA  TRANSFORMING  DEFENSE 


Characteristics  of  Good  Metrics  for  Performance  Based 
Logistics 


Presenter:  Kenneth  Doerr,  Associate  Professor,  Operations  Management,  Graduate 
School  of  Business  &  Public  Policy,  Naval  Postgraduate  School 

Presenter:  Donald  R.  Eaton,  RADM,  USN  (ret.),  Senior  Lecturer,  Arthur  Chair, 
Graduate  School  of  Business  &  Public  Policy,  Naval  Postgraduate  School 

Presenter:  Ira  A.  Lewis,  Associate  Professor  Graduate  School  of  Business  &  Public 
Policy,  Naval  Postgraduate  School 


Abstract 

Performance  Based  Service  Acquisition  is  a  Department  of  Defense  (DoD)  acquisition 
reform  that  has  had  noted  success  in  reducing  cost  and  streamlining  the  management  of  non¬ 
core  government  service  capabilities  (Office  of  the  Deputy  Under  Secretary  of  Defense  - 
Defense  Acquisition  Reform  [OSD-DAR],  2000).  The  guiding  principle  in  Performance  Based 
Service  Acquisition  is  that  when  an  outside  vendor  exists  who  can  perform  a  service  more 
effectively  than  a  government  user  could  organically,  that  user  should  specify  measurable 
outcomes  to  a  service  vendor,  and  allow  the  vendor  to  best  determine  the  appropriate 
processes  (how)  of  delivering  the  service.  Performance  Based  Logistics  (PBL)  is  an  extension 
of  Performance  Based  Service  Acquisition  aimed  at  the  logistic  services  for  major  weapon 
systems.  PBL  is  intended  to  reduce  lifecycle  cost,  increase  readiness,  improve  reliability  and 
reduce  the  logistical  footprint  of  weapon  systems. 

Introduction 

Performance  Based  Service  Acquisition  is  a  Department  of  Defense  (DoD)  acquisition 
reform  that  has  had  noted  success  in  reducing  cost  and  streamlining  the  management  of  non¬ 
core  government  service  capabilities  (Office  of  the  Deputy  Under  Secretary  of  Defense  - 
Defense  Acquisition  Reform  [OSD-DAR],  2000).  The  guiding  principle  in  Performance  Based 
Service  Acquisition  is  that  when  an  outside  vendor  exists  who  can  perform  a  service  more 
effectively  than  a  government  user  could  organically,  that  user  should  specify  measurable 
outcomes  to  a  service  vendor,  and  allow  the  vendor  to  best  determine  the  appropriate 
processes  (how)  of  delivering  the  service.  Performance  Based  Logistics  (PBL)  is  an  extension 
of  Performance  Based  Service  Acquisition  aimed  at  the  logistic  services  for  major  weapon 
systems.  PBL  is  intended  to  reduce  lifecycle  cost,  increase  readiness,  improve  reliability  and 
reduce  the  logistical  footprint  of  weapon  systems.  A  number  of  case  studies  of  successful  PBL 
initiatives  are  available  (e.g.,  Candreva,  et  al.,  2001). 

This  paper  takes  for  granted  the  success  of  PBL  initiatives,  and  takes  as  its  starting  point 
the  question  of  how  best  to  measure  the  degree  of  that  success.  In  support  of  our  prescriptions 
for  measurement,  we  will  draw  not  only  on  successful  best  practice,  but  also  on  the  underlying 
logic  and  justification  of  outsourcing,  as  laid  out  in  the  economics  and  management  literature. 
While  PBL  prescriptions  from  OSD  are  always  careful  to  explain  that  a  PBL  initiative  may  result 
in  the  selection  of  an  ‘organic  contractor’  (i.e.,  another  DoD  command),  actual  instances  of 
‘organic  contractors’  are  fairly  rare,  and  in  any  event,  some  of  the  same  measurement  issues 
arise  regardless  of  the  blend  of  private  sector  and  organic  resources. 
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While  measuring  the  performance  of  ongoing  PBL  initiatives  is  our  starting  point,  we  also 
intend  this  paper  to  inform  valuation  questions.  From  the  initial  question  of  whether  to  bring 
forward  a  weapon  system  or  a  major  component  of  a  weapon  system  as  a  candidate  for  PBL,  to 
later  design  questions  of  ‘what  form’  of  PBL  is  best  applied  to  that  candidate,  measurement 
issues  are  endemic.  After  all,  the  logistic  services  to  be  outsourced  will  be  priced  contractually, 
and  for  some  services,  there  is  no  clear  market  to  determine  that  price.  When  discussing 
CONUS  transportation,  prices  are  perhaps  not  difficult  to  determine  by  reference  to  a  market. 
However,  when  discussing  something  like  intermediate-level  maintenance  of  a  deployed 
weapon  system  on  which  the  DoD  has  a  monopsony  and  the  number  of  qualified  bidders  is 
quite  limited  (and  may  indeed  be  only  one  or  two),  the  market  paradigm  clearly  breaks  down, 
and  is  perhaps  best  understood  in  the  context  of  game  theory  (Shubik  &  Levitan,  1980).  Unlike 
the  simple  solutions  of  monopolistic  games,  however,  the  monopsonistic  game  of  buying 
weapon  systems  logistics  is  hampered  by  the  difficulty  of  measuring  the  value  of  the  services  to 
be  obtained. 

In  discussing  whether  a  case  could  be  made  for  the  privatization  of  a  particular 
governmental  service,  Bendick  (1984)  said  it  was  important  to  compare  private  to  ‘nonmarket’ 
(i.e.,  organic)  alternatives,  and  that  the  private  sector  should  only  be  employed  if  it  could 
reasonably  be  expected  to  be  more  efficient.  He  listed 

“four  aspects  of  market  efficiency  [that]  are  important  to  examine: 

•  In  producing  the  services  ...  do  the  private  sector’s  production  processes  and 
input  costs  allow  it  to  generate  output  at  a  lower  total  cost  than  could  the  public 
sector? 

•  Are  the  administrative  costs  incurred  by  government  to  mobilize  and  control  the 
private  sector  les  than  the  cost  savings  from  more  efficient  production? 

•  Is  the  supply  side  of  the  market  sufficiently  responsive  that  private  firms  enter 
markets  rapidly  and  smoothly? 

•  Are  purchasers  sufficiently  rational  and  careful,  and  the  quality  of  the  service 
sufficiently  definable  and  measurable,  that  effective,  informed  consumer 
sovereignty  can  be  exercised?  (Bendick,  1984,  pp.  153-154).” 

Each  of  these  considerations  is  potentially  problematic  when  examining  PBL  initiatives. 
When  considering  the  first  of  his  factors,  the  existence  of  PBL  contracts  in  which  the  private 
sector  vendor  has  hired  back  organic  resources  as  subcontractors  to  do  the  touch  labor  puts  in 
question  exactly  what  services  are  being  outsourced  -  logistics  or  management?  When 
considering  the  third  of  his  factors,  the  consolidation  of  the  defense  industry  and  the  decline  of 
the  number  of  independent  companies  that  might  act  as  potential  bidders  raise  concern. 
However,  this  paper  will  primarily  concern  itself  with  the  second,  and  especially  the  fourth  of  his 
factors.  We  will  discuss  how  an  excess  of  measurement  can  make  administration  of 
comprehensive  PBL  contracts  more  costly,  while  the  difficulty  with  defining  and  measuring  some 
logistic  services  make  consumer  sovereignty  difficult  to  establish. 

The  rest  of  the  paper  will  be  organized  as  follows.  First,  we  will  lay  out  a  structural 
framework  upon  which  measurement  issues  will  be  developed.  Upon  that  framework,  we  will 


CHARTING  A  COURSE  FOR  CHANGE: 

ACQUISITION  THEORY  AND  PRACTICE  FORA  TRANSFORMING  DEFENSE  -67- 


then  develop  questions  about  how  measurement  informs  which  sorts  of  candidates  are  best 
suited  for  PBL.  Finally,  we  will  discuss  how  measurement  issues  should  be  considered  when 
deciding  on  the  form  of  PBL  to  be  adopted  for  a  particular  candidate,  and  the  management  of 
ongoing  PBL  contracts.  We  are  not  attempting  to  clearly  delineate  between  good  and  bad 
measures,  or  good  or  bad  candidates  for  PBL.  Rather,  we  are  attempting  to  surface  imbedded 
measurement-related  issues  that  may  make  the  difference  between  a  problematic 
implementation  and  an  easy  one.  Thus,  this  paper  is  not  intended  as  a  guidebook  for 
implementation,  but  rather  as  a  guidebook  for  further  investigation. 

A  Hierarchical  Bridge  Framework  of  Measurement  for  PBL 

When  describing  logistics  service  acquisition  for  a  weapon  system  as  an  economic 
game,  it  is  important  at  the  outset  to  note  the  dissimilarity  between  the  two  players.  The  vendor 
has  a  clear  objective  of  maximizing  the  wealth  of  their  owners,  and  a  clear  profit  incentive 
(again,  we  assume  throughout  the  paper  that  we  are  dealing  with  a  private-sector  vendor).  The 
objective  of  the  user  acquiring  the  service  is  not  so  easy  to  state,  and  far  more  difficult  to 
assess.  Maximizing  national  security  would  be  one  way  to  state  the  objective,  and  the  incentive 
(at  least  at  the  organizational  level)  might  be  understood  in  the  same  terms  -  to  gain  more 
security  for  the  nation.  At  the  outset  then,  the  game  has  a  measurement  and  a  translation 
problem  -  measuring  the  services  in  terms  of  their  contribution  to  the  objectives  and  incentives 
of  the  DoD,  and  translating  that  measure  into  the  dollar  measurement  used  by  the  private 
sector. 


Of  course,  it  might  be  claimed  that  business  does  not  really  have  such  clear  objectives 
and  incentives  either.  There  is  a  venerable  literature  pointing  out  the  maximization  of 
shareholder  wealth  should  not  be  (and  is  not  in  practice)  the  sole  aim  of  a  public  corporation. 
Stakeholder  analysis  has  its  roots  in  this  observation  (Donaldson  &  Preston,  1995).  But  even 
stakeholder  analysis  (in  narrow  form  at  least)  does  not  deny  the  centrality  of  profit  as  a 
corporate  incentive,  rather  the  discussion  centers  on  rights  of  resource  holders,  and  equitable 
distribution  of  profits. 

The  management  fashion  of  Balanced  Scorecards  has  demonstrated  the  willingness  of 
corporate  executives  to  look  beyond  profit  in  analyzing  performance  (Kaplan  &  Norton,  1992). 
But  it  would  be  a  mistake  to  take  the  current  proliferation  of  Balanced  Scorecards  as  evidence 
that  corporations  suffer  under  the  same  sorts  of  fundamental  measurement  problems  with  their 
objectives  and  incentives  as  the  DoD.  The  Balanced  Scorecard  is  clearly  meant  to  be  a 
diagnostic  tool  to  inform  management  decisions  beyond  retrospective  financial  figures  about  the 
long  term  viability  of  the  firm  (i.e.,  it  is  meant  in  part  to  help  predict  and  control  future  financial 
performance).  Kaplan  and  Norton  (1992)  discuss  the  shortcomings  of  financial  performance 
measures  in  terms  of  their  ability  to  guide  (1)  the  innovation  necessary  to  obtain  future 
profitability,  (2)  the  diagnosis  of  internal  process  problems  that  limit  current  and  future 
profitability  and  (3)  the  relationship  with  the  customer  necessary  to  sustain  future  profitability. 
Their  main  criticisms  of  current  financial  measures  (which  are  a  part  of  the  Balanced  Scorecard) 
are  that  they  are  historical  and  external  to  operations.  They  tell  a  firm  how  well  it  has 
performed,  not  why,  or  what  to  do  next  to  maintain  or  improve  future  performance. 

But  measurement-related  differences  between  the  DoD  and  the  corporate  world  exist  not 
only  in  the  incentives  and  objectives  of  each,  but  also  in  the  process  capabilities  that  are 
important  in  developing  logistics  tactics  to  meet  those  objectives.  In  reviewing  essential 
dimensions  to  be  considered  in  logistic  performance  analysis  in  the  commercial  sector,  Mentzer 
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&  Konrad  (1991)  developed  a  matrix  in  which  five  core  logistics  functions  (transportation, 
warehousing,  inventory,  order  processing  and  administration)  could  be  measured  along  six 
dimensions  (cost,  labor,  facilities,  equipment,  time  &  energy).  Distinguish  between  those 
dimensions  and  the  four  “overarching  goals  of  PBL  ...  to  compress  the  supply  chain,  eliminate 
non-value  added  steps,  reduce  Total  Ownership  Cost  and  improve  readiness  for  weapons 
systems... (Department  of  Defense  -  Defense  Contract  Management  Agency  [DoD-DCMA], 
undated)”  to  which  one  should  add  “increased  reliability  and  reduced  logistics  footprint  (Office  of 
the  Secretary  of  Defense  [OSD],  2003)”.  Aside  from  cost,  these  sets  of  six  factors  seem  to  have 
little  in  common.  But  all  of  the  commercial  sector  factors  can  be  translated  into  cost,  and  can  be 
understood  as  the  essential  dimensions  that  must  be  managed  efficiently  and  effectively,  in 
order  to  facilitate  logistics  support  of  the  firm’s  profitability  objective.  The  DoD  factors,  on  the 
other  hand,  do  not  all  translate  so  readily  into  cost,  and  fall  into  three  categories  of  dimensions 
that  logistics  improves  warfighting  capability:  improved  readiness  (facilitated  both  directly  by  a 
focus  on  readiness  and  indirectly  by  a  focus  on  reliability),  increased  agility  (reducing  logistical 
footprint,  eliminating  non-value-added  steps,  supply  chain  compression,  and  improved 
reliability)  and  reducing  cost  (by  freeing  capital  for  other  warfighting  priorities). 

This  is  a  significant  difference  in  how  logistics  is  viewed.  The  concept  of  readiness 
shows  up  as  ‘equipment’  to  commercial  firms,  who  view  the  maintenance  and  functioning  (and 
depreciation)  of  their  operating  capital  primarily  as  a  financial  question  -  when  will  it  become  so 
expensive  to  maintain  that  I  will  have  to  replace  it?  Since  DoD  weapon  systems  are  often  quite 
old,  very  expensive  and  difficult  to  re-capitalize  (lacking  a  depreciation  mechanism, 
recapitalization  is  often  driven  by  technological  obsolescence),  readiness  is  a  much  more 
central  issue.  Improvements  in  readiness,  of  course,  improve  warfighting  capability;  but 
marginal  improvements  are  quite  difficult  to  value  in  dollar  terms.  The  idea  of  ‘agility’  is 
increasingly  important  to  commercial  firms,  but  agility  in  a  commercial  operation  means,  for 
example,  the  flexibility  to  quickly  change  production  volumes,  or  quickly  changing  production 
technology.  It  shows  up  in  the  list  above  as  ‘time’,  which  is  also  translatable  to  dollars.  DoD 
operations  on  the  other  hand  are  mobile,  and  mobility  directly  impacts  their  effectiveness. 

Agility  is  not  a  newly  discovered  competitive  dimension  -  it  has  always  been  an  operational 
necessity.  Once  again,  however,  the  operational  effectiveness  derived  from  a  marginal 
improvement  in  logistics  agility  is  very  difficult  to  translate  into  dollars. 

These  differences  in  organizational  objectives  and  the  consequent  logistics  objectives 
further  devolve  into  differences  in  process  measurement.  Caplice  &  Sheffi  (1994),  in  a 
classification  and  review  of  corporate  logistic  process  metrics  develop  three  categories: 
utilization,  productivity,  and  effectiveness  (see  Table  1).  Utilization  measures  simply  address 
the  question  of  how  much  of  a  resource  is  used,  compared  to  what  has  been  made  available. 
While  these  sorts  of  measures  may  be  useful  in  assessing  the  efficiency  of  a  narrow  segment  of 
a  process  (e.g.,  space  utilization  may  be  useful  in  assessing  the  efficiency  of  a  facilities  layout 
manager),  they  have  virtually  no  contribution  to  the  understanding  of  logistics  contribution  to 
organizational  objectives,  primarily  because  they  do  not  measure  outputs  at  all.  It  might  be 
claimed  that  they  measure  waste,  but  even  this  is  not  true  -  all  they  measure  is  activity,  not 
whether  that  activity  is  directed  toward  some  valued  outcome.  What  Caplice  and  Sheffi  (1994) 
have  called  effectiveness  measures,  on  the  other  hand,  ‘beg  the  question’  in  an  essential  way  - 
those  measures  are  only  as  good  as  the  norms  one  establishes  for  outputs.  They  may  be 
useful  for  historical  comparison  of  a  single  process,  but  their  value  in  comparing  across 
processes  or  in  guiding  resource  allocation  decisions  is  quite  limited.  Productivity  measures,  on 
the  other  hand,  incorporate  both  outputs  and  inputs.  For  the  corporation,  assessing  the 
contribution  of  an  activity  to  its  objectives  is  a  matter  of  relating  those  inputs  outputs  to  profits. 
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While  of  course  this  is  not  necessarily  easy  (e.g.,  single  factor  productivity  measures  do  not 
capture  a  comprehensive  cost  picture),  at  least  the  examples  given  by  Caplice  &  Sheffi  (1994) 
can  be  measured  or  translated  to  dollars  (e.g.,  dollars  paid  for  orders  processed,  or  shipments 
made),  and  this  is  broadly  true  of  metrics  proposed  in  other  reviews  of  corporate  logistic 
performance  measurement  systems  as  well  (e.g.,  Chow,  Heaver  &  Henriksson,  1994;  Lambert 
&  Burduroglu,  2000;  Mentzer  &  Konrad,  1991),  with  the  important  exception  of  customer 
satisfaction  metrics.  The  importance  of  the  ‘customer  view’  has  already  been  mentioned  in 
relation  to  Balanced  Scorecards,  is  often  mentioned  my  authors  on  logistic  performance 
measurement.  However,  it  is  worth  noting  that  Lambert  &  Burduroglu  (2000)  list  “reliance  on 
management  outside  of  logistics  to  identify  the  impact  [of  customer  satisfaction]  on  revenues, 
which  typically  does  not  happen”  as  a  primary  disadvantage  of  customer  satisfaction 
measurement.  Hence,  beyond  simple  utilization  measures,  corporate  logistic  performance 
measures  can,  or  are  desired  to  be,  understood  in  terms  of  their  impact  on  profitability. 


Table  1.  Corporate  Logistics  Metrics  (from  Caplice  and  Sheffi,  1994) 


Dimension 

Form  of  Metric 

Logistics  Examples 

Utilization 

Actual  Input  / 

Normed  Input 

Labor  hour  used  /  labor  hours  budgeted 

Area  of  warehouse  occupied  /  total  area 

Hours  machine  used  /  machine  capacity 

Productivity 

Actual  Output  / 
Actual  Input 

Ton-miles  delivered  /  costs  incurred 

Orders  processed  /  hours  of  labor 

Pallets  unloaded  /  hour  of  dock  time 

Effectiveness 

Actual  Output  / 

Norm  Output 

Items  filled  /  items  requested 

Shipments  on  time  /  shipments  sent 

Transactions  w/o  error  /  total  transactions 

Compare  those  corporate  logistic  measures  to  what  might  be  proposed  as  a  productivity 
ratio  for  weapons  systems  logistics  - 

Operational  Availability  (Ao)  of  deployed  systems  /  Total  deployed  hours. 

At  first  glance,  this  looks  like  a  utilization  measure,  not  a  productivity  measure  -  but  Ao 
is  often  used  as  a  surrogate  for  readiness,  which  is  typically  given  as  a  primary  outcome 
objective  of  military  logistics  (it  would  be  a  utilization  measure  if,  e.g.,  flight  hours  were  in  the 
numerator).  The  denominator  translates  to  dollars  in  a  budget  (whether  or  not  they  could  be 
translated  to  an  actual  cost  is  another  issue).  But  the  numerator  is  not  and  should  not  be 
translatable  to  dollars,  because  profit  is  not  the  objective.  While  measurable,  it  is  difficult  to 
value  in  terms  of  the  dollars  that  might  be  spent  to  increase  it,  or  relinquished  in  order  to  pursue 
other  priorities.  Another  problem  is  that  Ao  is  only  a  surrogate  for  readiness  because  it  is  a 
‘single  factor’  measure.  It  is  also  not  fine  grained  enough  for  many  resource  allocation  decision 
we  wish  to  make  (hence  the  distinction  between  ‘mission  capable’  and  ‘fully  mission  capable’ 
systems).  And  finally  readiness  itself,  after  all,  is  only  a  surrogate  for  the  organizational 


CHARTING  A  COURSE  FOR  CHANGE: 

ACQUISITION  THEORY  AND  PRACTICE  FORA  TRANSFORMING  DEFENSE  -  70  - 


objectives  of  the  DoD  (i.e.,  ready  for  what?).  Note  that  if  Ao  were  really  the  objective,  it  could  be 
maximized  by  parking  equipment.  Hence,  logistic  performance  of  weapon  systems  are  more 
difficult  to  measure  than  commercial  logistics  (at  least  in  terms  of  productivity),  and  perhaps 
more  attenuated  from  DoD  objectives  than  are  commercial  logistic  measurement  systems. 

Figure  1 .  A  hierarchical  framework  of  measurement  issues 


How  do  these  measurement  issues  inform  the  decision  to  bring  forward  a  weapon 
system  or  component  as  a  PBL  Candidate?  First,  again  considering  only  outsourced  PBL 
solutions,  we  must  consider  the  economic  logic  behind  outsourcing.  One  basic  economic 
justification  of  outsourcing  is  the  tradeoff  of  economies  of  scale  with  reduction  in  transaction 
costs.  If  the  outsourced  service  can  be  performed  by  an  organization  that  offers  similar  services 
to  a  number  of  other  customers,  that  organization  gains  economies  of  scale,  and  should  be  able 
to  offer  the  service  more  cheaply  than  if  it  were  done  by  the  outsourcing  organization  in  house. 
The  price  that  is  usually  paid  for  such  outsourced  services  is  usually  in  terms  of  increased 
transaction  costs  to  negotiate  price  and  services,  and  monitor  performance  (Aubert,  Rivard  & 
Patry,  1996).  When  economies  of  scale  are  difficult  to  obtain,  as  with  a  unique  weapon  system 
requirement,  some  of  the  underlying  justification  for  outsourcing  disappears.  On  the  other  hand, 
high  internal  transaction  costs,  due  to  e.g.,  high  reporting  requirements,  or  inefficient  internal 
controls  make  outsourcing  relatively  more  attractive.  If  high  internal  transaction  costs  are  part  of 
the  justification  for  outsourcing  a  PBL  contract,  then  it  is  important  that  the  system  or  component 
being  outsourced  avoid  some  of  those  transaction  costs.  When  measurement  of  logistic 
outcomes  (readiness,  agility  and  cost)  is  more  difficult,  it  will  mean  higher  transaction  costs, 
because  performance  monitoring  systems  will  have  to  be  more  elaborate,  and  fair  prices  will  be 
more  difficult  to  determine  and  negotiate. 


One  way  to  make  pricing  and  performance  monitoring  easier  is  by  reference  to  a  market 
for  similar  services.  Hence,  in  prescribing  a  methodology  for  the  analysis  of  performance  based 
contracts  for  contract  managers,  market  research  is  indicated  as  a  required  step  (OSD-DAR, 
2000).  For  comprehensive  weapon  system  logistical  support,  or  for  weapon  system-unique 
components,  there  will  likely  be  no  ready  market  for  maintenance,  or  many  other  logistical 
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support  functions.  In  those  cases,  the  implementation  of  an  outsourced  PBL  solution  will 
require  more  cost  and  effort  to  develop  appropriate  metrics,  and  negotiate  appropriate  prices. 

In  summary,  measurement  issues  are  endemic  to  the  relationship  between  commercial 
sector  vendors  and  the  DoD.  From  the  point  of  view  of  measurement,  the  best  PBL  candidates 
are  those  with  external  markets  for  services,  and  clear  outcomes  that  are  easy  to  relate  to 
mission  objectives.  This  is  not  to  place  a  definite  boundary  on  the  systems  where  PBL  ought  to 
be  applied,  but  only  to  point  out  that  measurement  issues  may  make  some  PBL 
implementations  far  more  difficult  and  expensive,  and  may  affect  the  form  of  the  PBL  solution. 

Measurement,  the  PBL  support  spectrum  and  the  management 
of  ongoing  contracts 

One  of  the  characteristics  of  PBL  is  that  general  characterizations  are  hard  to  make. 

The  top  level  guidance  for  the  initiative  always  has  caveats  such  as 

“there  is  no  one-size-fits-all  approach  to  PBL.  Several  programs  have  started  the  move 
to  PBL  under  initiatives  designed  to  meet  the  programs’  specific  requirements.  Each 
program  has  tailored  the  PBL  application  to  its  unique  circumstances  taking  into  account 
cost,  schedule,  or  product  integrity  to  meet  warfighter  capability.  (DoD,  2001,  p.  2-2)” 

In  reviewing  implementations,  a  wide  variety  of  approaches  can  be  found,  in  terms  of 
measurement  and  incentives,  and  in  terms  of  the  level  at  which  the  PBL  contract  is  written: 
from  a  complete  weapon  system  as  with  the  DDG-51 ,  to  component  level  stock  support,  as  with 
the  AEGIS.  The  spectrum  of  choices  is  usually  described  in  terms  of  the  degree  of  commercial 
support  involved,  and  a  frequently  encountered  graphic  (which  we  have  been  unable  to  track  to 
its  original  source)  is  shown  in  Figure  2. 

Figure  2.  The  PBL  Support  Spectrum. 
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While  examples  of  systems  are  often  given  in  association  with  this  chart,  and  definitions 
of  the  various  anchor  points  (e.g.,  Mini-Stock  Point)  are  offered,  very  little  guidance  can  be 
found  for  the  program  manager  as  to  what  characteristics  of  a  weapon  system  should  inform  the 
choice  of  the  degree  of  commercial  involvement,  and  whether  the  contract  should  be  offered  at 
the  system,  or  component  level.  It  is  our  position  that  measurement  issues  should  inform  this 
choice. 
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A  primary  aspect  of  measurement  informing  the  choice  of  commercial  involvement, 
which  we  have  not  discussed  yet,  is  risk.  As  should  be  clear  by  reference  to  Figure  1 ,  vendors 
will  be  primarily  interested  in  reductions  of  financial  risk,  while  the  DoD  is  entirely  concerned 
with  operational  risk.  The  tradeoff  of  these  two  kinds  of  risk  is  central  to  the  logic  of  PBL 
outsourcing.  Contracts  are  almost  always  offered  across  multiple  years  (lowering  financial  risk 
for  the  vendor),  with  the  expectation  that  the  vendor  will  assume  some  degree  of  operational 
risk.  Figure  3  shows  the  expected  tradeoff  of  operational  risk. 

Figure  3.  Intended  risk  transfer  under  PBL.  From  DoD-DCMA  (undated). 


Although  risk  is  clearly  indicated  as  a  factor  to  consider  when  developing  a  PBL  strategy 
(ASN-RDA,  2003;  Office  of  the  Assistant  Deputy  Under  Secretary  of  Defense  -  Logistics  & 
Material  Readiness  [OSD-LMR],  Undated),  this  factor  is  rarely  mentioned  as  a  candidate  for 
measures  of  ongoing  performance  in  PBL  contracts.  Indeed,  it  has  been  said  that 

“minimal  contract  management  involvement  is  anticipated  as  long  as  the  contractor 
meets  contractually  specified  performance  metrics.  However  our  involvement  may 
increase  if  the  contractor  systems  and  processes  are  not  functioning  correctly  and  end 
users  are  not  appropriately  supported  (DoD-DCMA,  undated,  pp  28-29. )” 

This  is  a  curious  form  of  risk  transfer.  Operational  risk,  as  we  understand  it,  involves  the 
variance  associated  with  outcomes.  The  assumption  of  operational  risk  by  a  contractor  then 
would  entail  accountability  for  control  of  that  variance,  and  assurance  (with  appropriate  remedy 
in  the  case  of  failure)  of  the  mitigation  of  its  impact.  Here,  rather,  it  seems  that  vendors  are 
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being  asked  to  assume  some  aspects  of  risk  (not  clearly  defined),  but  that  the  DoD  will  ‘increase 
involvement  if  the  contractor’  fails  to  meet  requirements.  We  do  not  claim  that  this  is  risk 
transfer  in  name  only,  but  that  the  form  of  risk  being  assumed  by  vendors  is  not  what  is 
commonly  understood  as  operational  risk,  and  the  degree  of  risk  they  are  assuming  is 
apparently  quite  limited. 

It  is  our  view  that  the  degree  of  operational  risk  a  contractor  can  assume  is  limited  in 
many  cases  by  the  nature  of  our  operations.  It  is  unrealistic,  for  example,  to  assume  that 
contractors  will  be  able  to  perform  operational  level  maintenance  on  a  ground  combat  weapon 
system;  difficult  issues  relating  to  the  physical  risk,  insurance,  and  liability  of  non-DoD  personnel 
in  or  near  combat  need  to  be  addressed.  These  sorts  of  operational  risk  are  difficult  to 
measure,  and  even  more  difficult  to  value.  We  think  it  likely  therefore,  that  commercial  sector 
vendors  will  be  reluctant  to  undertake  it.  Depot  level  maintenance,  and  operation  of  CONUS 
inventory  control  points  involve  less  operational  risk,  and  risk  in  a  form  that  is  easier  to  measure, 
and  less  costly  for  a  vendor  to  assume,  and  hence  we  think  it  likely  that  the  more  operational 
risk  involved  in  the  logistical  support  of  a  particular  system,  the  more  organic  resources  will 
need  to  be  involved. 

Proposition  1.  When  operational  risk  is  high  or  difficult  to  measure,  PBL 
strategies  should  seek  less  commercial  sector  involvement. 

Within  the  context  of  a  price  negotiation,  it  is  also  key  to  understand  the  benefit  we 
provide  by  eliminating  financial  risk,  as  this  is  part  of  what  we  are  paying  to  potential  vendors. 
Especially  if  interest  rates  and  rise  and  the  difference  between  the  cost  of  capital  and  risk-free 
rates  increase,  what  the  DoD  offers  in  terms  of  financial  risk  mitigation  is  highly  valuable.  This 
valuable  benefit  is  not  free  for  the  government  to  offer,  and  should  be  incorporated  into  pricing 
and  contract  negotiations.  If  less  operational  risk  is  assumed  by  the  vendor  (or  if  that  risk  is 
difficult  to  assess),  less  financial  risk  should  be  mitigated  -  meaning  contract  terms  should  be 
reduced. 

Proposition  2.  When  commercial  sector  vendors  assume  less  (measurable) 
operational  risk  under  a  PBL  contract,  the  term  of  that  contract  should  be  less. 

On  the  other  hand,  the  outcomes  of  PBL  strategies  involving  only  certain  components,  or 
only  depot-level  support,  are  more  difficult  to  tie  to  weapon  system  outcomes.  Consider  Figure 
4,  which  shows  a  highly  stylized  and  simplified  version  of  a  weapon  system  and  its  major 
components,  along  with  the  failure  rates  (mean  time  to  failure)  of  each  of  the  components. 
Assuming  failure  of  any  of  the  components  cause  the  weapon  system  to  become  non-mission 
capable,  the  failure  rate  of  the  overall  weapon  system  is  then  an  order  statistic,  formed  of  the 
distributions  of  the  time  to  failure  of  all  of  the  components.  Now  consider  the  problem  faced  by 
a  program  manager  who  has  decided  that  his  best  PBL  strategy  involves  outsourcing  only 
component  A  (the  one  with  the  highest  failure  rate).  To  properly  value  the  impact  of,  for 
example  a  proposed  incentive  to  improve  the  reliability  of  component  A  by  10%,  the  program 
manager  would  need  not  only  distributional  information  about  the  time  to  fail  of  all  the  other 
components,  but  also  a  working  model  which  imbeds  that  system  in  mission  requirements. 

After  all,  the  final  value  of  an  improvement  in  reliability  of  a  component  (to  readiness  -  of  course 
there  are  other  benefits  in  terms  of  reduced  life-cycle  cost  of  spares,  and  improved  agility 
through  reduced  footprint)  rests  in  the  increased  likelihood  of  mission  success  in  the  deployed 
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weapon  system.  The  sort  of  integrated  simulation  model  needed  to  properly  assess  the  impact 
of  improved  component  reliability  would  be  expensive,  and  more  importantly,  time  consuming  to 
build.  Given  the  time  pressures  put  upon  Program  Managers,  it  is  easy  to  see  that  the  situation 
is  problematic. 

Proposition  3.  PBL  strategies  involving  less  than  comprehensive  logistical 
support  of  a  weapon  system  (e.g.,  for  a  component)  should  nonetheless  have 
integrated  weapons  system  models  in  support  of  their  business  case  analysis. 

In  summary,  measurement  issues  exist  across  the  PBL  spectrum,  but  present  different 
sorts  of  challenges  at  either  end.  Ultimately  there  are  at  least  two  core  measurement  issues 
that  should  be  referred  to  when  deciding  on  an  appropriate  level  of  support  within  the  PBL 
spectrum.  The  first  is  the  valuation  of  outcome-related  performance,  and  the  second  is 
valuation  of  operational  and  financial  risk.  While  outcomes  are  easier  to  measure  at  the  right 
end  of  the  spectrum,  one  is  less  likely  to  find  a  relevant  market  to  support  price  and  value 
decisions.  On  the  left  hand  of  the  spectrum,  markets  may  well  exist  that  essentially  duplicate, 
for  example,  the  services  of  a  mini-stock  point.  However,  the  valuation  of  those  isolated 
services  in  terms  of  weapon  systems  performance  is  even  more  difficult. 

Figure  4.  Need  for  integrated  system  model  to  judge  impact  of  component  outcomes. 


Aside  from  risk,  the  main  distinction  we  will  develop  when  discussing  management  of 
ongoing  PBL  contracts  is  the  difference  between  process  and  outcome  measures.  It  is  our 
position  that,  while  PBL  is  clearly  intended  to  buy  outcomes,  and  relieve  management  of  the 
necessity  of  monitoring  the  details  of  ‘how’  performance  is  obtained,  a  great  deal  of  effort  is  still 
being  devoted  to  process  measurement.  It  has  recently  been  said  that  ‘too  many  metrics’  is  a 
major  problem  with  PBL  implementation  (Office  of  the  Secretary  of  Defense,  Undersecretary  for 
Acquisition,  Technology  &  Logistics  [OSD-ATL],  2004).  If  the  DoD  is  buying  outcomes,  not 
process,  then  it  may  be  that  much  of  the  process  measurement  is  unnecessary. 

An  example  of  ‘too  many  metrics’  is  shown  in  Figure  5.  This  is  a  slide  shown  in  a  brief  to 
a  base  commander  to  provide  an  overview  of  the  PBL  contracts  at  his  base.  This  is  a  small 
base,  with  only  a  handful  of  PBL  contracts.  Clearly,  the  commander  understands  that  there 
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are  too  many  metrics,  and  is  tracking  them  quarterly  in  order  to  push  for  their  reduction.  Here, 
the  number  of  metrics  itself  has  become  a  metric,  with  visibility  to  the  top  operating  officer  at  a 
command. 


Figure  5.  A  measure  of  PBL  measures  used  at  one  DoD  command. 
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Exactly  how  the  superabundance  of  metrics  arises  is  an  open  question.  PBL  is  a 
process  meant  to  streamline  managements  concern  with  the  details  of  a  logistical  process.  In 
part,  it  may  arise  from  a  broader  context  within  DoD,  of  understanding  the  systemic 
relationships  of  which  logistics  is  only  a  part.  Under  various  titles,  including  Integrated  Logistics 
Support,  the  last  several  years  has  seen  an  increasing  awareness  of  the  embedded  nature  of 
logistical  support,  and  the  inter-relationships  involved  between  e.g.,  manpower,  maintenance, 
and  supply.  Figure  6  is  taken  from  a  presentation  to  a  PBL  ‘tiger  team’  concerned  with 
establishing  metrics.  Figure  6a  lists  the  ‘balanced  scorecard’  of  top  level  factors  for  weapon 
system  support. 


Figure  6a.  DoD  Balanced  Scorecard  for  Weapon  System  Support.  From  PRTM  (2004). 
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Figure  6b,  translates  the  scorecard  into  logistics  measures,  by  mapping  between  the 
high  level  scorecard  factors,  and  the  primary  factors  (process  elements)  of  another 
management  fashion  called  Integrated  Logistical  Support  (ILS).  The  details  of  ILS  process 
elements  are  not  germane  here,  only  to  note  that  they  are  indeed  detailed  and  process  oriented. 
If  one  starts  with  a  multidimensional  balanced  scorecard,  and  works  through  these  process 
elements,  it  is  easy  to  see  how  the  number  of  metrics  that  must  be  tracked  could  be  numbered 
in  the  hundreds  -  in  fact,  it  would  be  hard  to  avoid.  One  must  ask  why  however,  if  we  are 
engaged  in  an  initiative  to  buy  performance,  we  are  starting  with  a  detailed  map  of  the  internals 
of  the  process.  Wouldn’t  it  make  more  sense  to  measure  only  key  outcomes,  and  measure 
them  well? 

When  we  first  presented  this  idea  at  a  conference,  we  were  met  with  the  objection  that 
an  abundance  of  measures  do  not  necessarily  distract  a  decision  maker  from  key  tasks.  The 
analogy  was  drawn  to  a  pilot  in  a  jet,  where  the  cockpit  has  a  superabundance  of  meters  and 
instruments,  almost  all  of  which  can  be  ignored,  except  in  the  case  of  an  emergency.  The 
analogy  is  a  telling  one,  in  that  most  of  the  people  making  decisions  about  metrics  for  PBL  have 
themselves  been  pilots,  or  ship  captains,  or  in  charge  of  some  complex  process  in  the  past. 
However,  PBL  is  not  supposed  to  present  the  DoD  with  a  complex  process  to  manage  -  it  is 
supposed  to  take  one  off  the  hands  of  the  DoD.  We  aren’t  supposed  to  be  flying  the  plane  -  we 
are  supposed  to  be  passengers.  When  you  are  paying  someone  else  to  get  you  to  your 
destination,  you  care  about  the  price  of  the  ticket,  and  arriving  on  time. 
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Of  course  this  is  a  simplification.  When  discussing  a  PBL  contract  with  one  deployed 
squadron,  we  were  met  with  a  complaint  about  the  difficulty  of  obtaining  requisition  status  for 
parts  that  were  backordered.  The  maintenance  officer  for  that  squadron  was  not  impressed  that 
the  average  lead  time  to  get  the  parts  had  been  reduced,  because  his  primary  decision  when 
confronted  with  a  backorder  was  whether  or  not  to  devote  the  manpower  to  strip  a  spare  part 
from  another  downed  aircraft.  It  was  a  decision  that  was  difficult  to  make,  without  knowing  how 
long  it  would  be  before  the  spare  part  arrived.  Clearly,  in  this  case,  some  visibility  into  the 
process  was  needed,  but  only  because  a  key  operational  decision  rested  on  the  measurement 
of  that  process.  Given  that  at  least  part  of  the  justification  for  PBL  is  meant  to  be  a  reduction  in 
transaction  costs,  we  think  this  should  probably  be  a  general  rule. 


Proposition  4.  In  the  management  of  ongoing  PBL  contracts,  metrics  should 
primarily  concern  themselves  with  valued  outcomes,  and  should  be  related  to 
weapon  system  cost,  readiness,  and  agility.  Process  measures  should  only  be 
applied  when  key  operating  decisions  depend  on  the  status  of  the  process  itself. 


Figure  6b.  Metric  Areas  informed  by  Balanced  Scorecard,  and  ILS  Process  Elements.  From 
PRTM  (2004). 
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Finally,  we  turn  to  the  measurement  of  risk  in  ongoing  PBL  contracts.  Operational  risk  is 
always  difficult  to  assess.  In  the  context  of  support  for  a  weapon  system,  it  can  be  understood 
as  variance  in  the  logistics-related  readiness  of  that  system.  A  common  measure  of  readiness 
is  Ao.  To  see  how  variability,  or  risk,  affects  Ao,  consider  Table  2.  The  table  shows  the 
availability  of  two  squadrons  of  10  aircraft  over  a  20  day  period.  Over  that  period,  both 
squadrons  would  report  operational  availability  of  95%.  But  consider  that  a  mission  requiring  10 
aircraft  would  be  degraded  (more  likely  to  fail)  only  5%  of  the  time  in  the  first  case,  but  50%  of 
the  time  in  the  second  case.  The  difference  is  the  variance  in  Ao.  To  our  knowledge,  in  spite  of 
the  exhortations  of  the  centrality  of  risk  assumption  and  readiness  to  PBL,  there  are  no 
programs  currently  tracking  this  measure. 
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Table  2.  Distributional  differences  in  a  95%  Ao. 
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In  a  very  thoughtful  document,  the  office  of  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition  (ASN-RDA,  2003)  listed  factors  to  consider  when 
deciding  whether  to  use  a  PBL  strategy:  life-cycle  stage,  acquisition  program  strategy  (including 
programmatic  risks),  organic  impact  (e.g.,  maintaining  engineering  expertise),  commercial  base 
(including  additional  risk  required  of  industry  partners),  design  considerations  (including  risk 
associated  with  incentives  and  performance  thresholds),  and  technology  considerations 
(including  supportability  risks).  Although  risk  is  mentioned  in  four  of  the  six  factors,  there  is  no 
mention  of  the  sort  of  operational  risk  discussed  above,  and  shown  in  Figure  3.  Indeed,  there  is 
little  mention  of  risk  for  ongoing  contracts  in  this,  or  any  other  guidance  documents  for  PBL.  We 
find  this  curious.  If  part  of  what  we  are  buying  is  operational  risk  mitigation  (in  key  performance 
dimensions),  it  seems  to  us  that  it  ought  to  be  measured. 

Propositions  5.  Operational  risk  (variability)  in  key  performance  measures  should 
be  tracked  for  ongoing  PBL  contracts.  Where  essential  to  mission  support,  a 
reduction  in  variability  should  be  supported  with  appropriate  incentives. 

In  summary,  this  paper  has  presented  a  framework,  and  propositions  about  the  impact  of 
measurement  on  PBL.  None  of  the  propositions  have  the  status  of  fully  supported  hypotheses, 
or  fully  developed  theorems.  All  need  further  investigation.  Some  of  the  propositions  are 
empirical,  and  need  to  be  investigated  in  the  field.  Others  are  prescriptive,  and  need  to  be 
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supported  by  modeling  and  analysis.  Our  hope  is  that  we  have  furthered  the  discussion  of 
metrics  for  PBL,  and  added  to  the  momentum  for  improved  implementations  of  PBL. 
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